OpenNMT Forum

Train onmt with multiple features on tensorflow v2.0

Hello Fellow Researchers,

I am trying to run onmt with multiple features.
My config yml file is as follows:

model_dir: transformer_feature

data:
  source_tokenization: space.yml
  target_tokenization: space.yml
  train_features_file: 
    - data/train.zh
    - data/train.zh_f
  train_labels_file: data/train.yue
  eval_features_file: 
    - data/vaild.zh
    - data/vaild.zh_f
  eval_labels_file: data/vaild.yue
  source_1_vocabulary: data/zh.vocab
  source_2_vocabulary: data/zh_f.vocab
  target_vocabulary: data/yue.vocab

train:
  save_checkpoints_step: 1000

eval:
  external_evaluators: BLEU

Can anyone help me where I am doing wrong and how can I use multiple features model.

onmt-main --model multi_features_nmt.py --config nmt_f.yml --auto_config train --num_gpus 1

The code of multi_source_nmt.py is as follows:

import tensorflow as tf
import opennmt as onmt

def model():
        return onmt.models.Transformer(
                source_inputter=onmt.inputters.ParallelInputter([
                        onmt.inputters.WordEmbedder(embedding_size=512),
                        onmt.inputters.WordEmbedder(embedding_size=8)],
                        reducer=onmt.layers.ConcatReducer()),
                target_inputter=onmt.inputters.WordEmbedder(embedding_size=512),
                num_layers=6,
                num_units=512,
                num_heads=8,
                ffn_inner_dim=2048,
                dropout=0.1,
                attention_dropout=0.1,
                ffn_dropout=0.1)

I got output with errors :

INFO:tensorflow:Creating model directory transformer_feature
INFO:tensorflow:Using parameters:
data:
  eval_features_file:
  - data/vaild.zh
  - data/vaild.zh_f
  eval_labels_file: data/vaild.yue
  source_1_vocabulary: data/zh.vocab
  source_2_vocabulary: data/zh_f.vocab
  source_tokenization: space.yml
  target_tokenization: space.yml
  target_vocabulary: data/yue.vocab
  train_features_file:
  - data/train.zh
  - data/train.zh_f
  train_labels_file: data/train.yue
eval:
  batch_size: 32
  external_evaluators: BLEU
infer:
  batch_size: 32
  length_bucket_width: 5
model_dir: transformer_feature
params:
  average_loss_in_time: true
  beam_width: 4
  decay_params:
    model_dim: 512
    warmup_steps: 8000
  decay_type: NoamDecay
  label_smoothing: 0.1
  learning_rate: 2.0
  num_hypotheses: 1
  optimizer: LazyAdam
  optimizer_params:
    beta_1: 0.9
    beta_2: 0.998
score:
  batch_size: 64
train:
  average_last_checkpoints: 8
  batch_size: 3072
  batch_type: tokens
  effective_batch_size: 25000
  keep_checkpoint_max: 8
  length_bucket_width: 1
  max_step: 500000
  maximum_features_length: 100
  maximum_labels_length: 100
  sample_buffer_size: -1
  save_checkpoints_step: 1000
  save_summary_steps: 100

2020-03-23 10:02:47.624856: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-03-23 10:02:47.737676: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:3d:00.0
2020-03-23 10:02:47.737998: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-03-23 10:02:47.740106: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-03-23 10:02:47.742240: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-03-23 10:02:47.742655: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-03-23 10:02:47.745393: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-03-23 10:02:47.747488: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-03-23 10:02:47.753415: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-03-23 10:02:47.766813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-03-23 10:02:47.767323: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2020-03-23 10:02:47.778892: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2100000000 Hz
2020-03-23 10:02:47.782749: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4ebefd0 executing computations on platform Host. Devices:
2020-03-23 10:02:47.782764: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
2020-03-23 10:02:48.529690: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x594b8f0 executing computations on platform CUDA. Devices:
2020-03-23 10:02:48.529718: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-PCIE-16GB, Compute Capability 7.0
2020-03-23 10:02:48.542907: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:3d:00.0
2020-03-23 10:02:48.543456: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-03-23 10:02:48.543472: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-03-23 10:02:48.543485: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-03-23 10:02:48.543497: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-03-23 10:02:48.543507: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-03-23 10:02:48.543517: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-03-23 10:02:48.543530: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-03-23 10:02:48.555279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-03-23 10:02:48.555336: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-03-23 10:02:48.564077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-23 10:02:48.564100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2020-03-23 10:02:48.564112: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2020-03-23 10:02:48.572619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14926 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0000:3d:00.0, compute capability: 7.0)
WARNING:tensorflow:No checkpoint to restore in transformer_feature
INFO:tensorflow:Training on 2364621 examples
INFO:tensorflow:Accumulate gradients of 9 iterations to reach effective batch size of 25000
2020-03-23 10:03:10.675534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:3d:00.0
2020-03-23 10:03:10.675708: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-03-23 10:03:10.675727: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-03-23 10:03:10.675745: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-03-23 10:03:10.675764: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-03-23 10:03:10.675779: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-03-23 10:03:10.675797: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-03-23 10:03:10.675816: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-03-23 10:03:10.689538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-03-23 10:03:10.689592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-23 10:03:10.689620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2020-03-23 10:03:10.689630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2020-03-23 10:03:10.697331: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/device:GPU:0 with 14926 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0000:3d:00.0, compute capability: 7.0)
2020-03-23 10:03:10.768500: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
INFO:tensorflow:Saved checkpoint transformer_feature/ckpt-0
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1781: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
2020-03-23 10:04:11.385303: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:143] Filling up shuffle buffer (this may take a while): 1236288 of 2364621
2020-03-23 10:04:20.818730: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:193] Shuffle buffer filled.
2020-03-23 10:04:21.192175: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Invalid argument: ConcatOp : Dimensions of inputs should match: shape[0] = [76,40,512] vs. shape[1] = [76,71,8]
         [[{{node cond_2/then/_44/transformer/parallel_inputter/concat}}]]
         [[Func/cond_2/then/_44/gradients/global_norm/write_summary/summary_cond/then/_1731/input/_1742/_442]]
2020-03-23 10:04:21.192282: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Invalid argument: ConcatOp : Dimensions of inputs should match: shape[0] = [76,40,512] vs. shape[1] = [76,71,8]
         [[{{node cond_2/then/_44/transformer/parallel_inputter/concat}}]]
Traceback (most recent call last):
  File "/usr/local/bin/onmt-main", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/opennmt/bin/main.py", line 189, in main
    checkpoint_path=args.checkpoint_path)
  File "/usr/local/lib/python3.6/dist-packages/opennmt/runner.py", line 207, in train
    export_on_best=eval_config.get("export_on_best"))
  File "/usr/local/lib/python3.6/dist-packages/opennmt/training.py", line 89, in __call__
    self._accumulate_next_gradients(dataset, report_steps=report_steps)):
  File "/usr/local/lib/python3.6/dist-packages/opennmt/data/dataset.py", line 433, in _fun
    outputs = _tf_fun()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 457, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 520, in _call
    return self._stateless_fn(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1823, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1141, in _filtered_call
    self.captured_inputs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1224, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 511, in call
    ctx=ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  ConcatOp : Dimensions of inputs should match: shape[0] = [76,40,512] vs. shape[1] = [76,71,8]
         [[{{node cond_2/then/_44/transformer/parallel_inputter/concat}}]]
         [[Func/cond_2/then/_44/gradients/global_norm/write_summary/summary_cond/then/_1731/input/_1742/_442]]
  (1) Invalid argument:  ConcatOp : Dimensions of inputs should match: shape[0] = [76,40,512] vs. shape[1] = [76,71,8]
         [[{{node cond_2/then/_44/transformer/parallel_inputter/concat}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference__tf_fun_45023]

Function call stack:
_tf_fun -> _tf_fun

Currently the CUDA version of my server is 10.0, and OpenNMT-tf==2.4.0 which depends on Tensorflow==2.0 .

Thanks & Regards

Hi,

Can you check that the 2 input files train.zh and train.zh_f are correctly aligned and that each line has the same number of tokens?